#KV cache compression

NVIDIA researchers developed Dynamic Memory Sparsification (DMS), a novel method that compresses KV caches by 8× in Transformer-based LLMs, improving inference efficiency while maintaining accuracy.